Summary of Some Statistical Issues

In terms of statistical ideas, I would make a very broad distinction between two prominent sets of problems at this conference. First there are a number of problems in which the main feature is a very large amount of data, requiring new methods and considerable computing power. An example that has already been used with success in astronomy is the use of false discovery rates in problems involving a great number of tests, and we heard here about new adaptations of wavelet and ridgelet techniques for identifying structure in images, about smoothing methods in multi-dimensional image processing, and new methods for on-line data mining. I won’t attempt to summarize this class of problems, although it is clearly very important, not only in physics and astronomy but in a number of scientific problems, especially including genomics, where there is very active development of statistical techniques. Another class of problems seems simpler (to a statistician) on a first reading. An example is independent Poisson counts from background events and possible signal events. We should not forget, though, that elaborate experimental techniques and considerable ingenuity in data processing, have preceded the presentation of a small amount of data. For this setting one would expect that standard statistical methods would provide a simple, and even a best, answer, but as we have seen even in this context this is not always the case. Certainly inference about the ratio of Poisson mean parameters is satisfactorily solved using the binomial likelihood. Statistical inference for the difference between two Poisson means is somewhat more difficult, as we have to rely on some approximate argument, and with small counts the usual normal approximations will not be reliable. in As Sir David Cox stressed in the panel discussion, the science of statistics develops most fruitfully in close collaboration with applications, and this problem is a good example of something that is indeed sufficiently specialized to the HEP context that it is not in the repertoire of ‘off-the-shelf’ statistical methods. Some general ideas which should inform the solution include the very important notion that confidence intervals, however developed, should have good properties in repeated observation of the same experimental system, even if these repetitions are hypothetical. In my view the definition of ‘same experimental system’ needs great care, in order to avoid difficulties similar to, but more subtle than, the problem of two measuring instruments discussed in Cox and mentioned in Cousins. Unfortunately it seems extremely difficult to ‘mathematize’ this notion; statisticians have spent many years of effort on the topic, and a single widely accepted solution has not emerged. At this time the best we can advise is to look at problems on a case by case basis. Likelihood methods are well accepted in the HEP community, but not always used in quite the same manner as used by statisticians. To clarify, suppose we have a single parameter model f(x; θ) and observe a sample x = (x1, . . . , xn) of independent observations from this model. The log-likelihood function `(θ; x) = log Πf(xi; θ) is a sum of n terms, and we can apply the central limit theorem to ∂`(θ; x)/∂θ to derive the following approximations: